Sensing Surfaces
Sensors that acquire 3D data are useful for many applications. For example, a system for automated ‘bin-picking’ in a factory can acquire 3D data as a precursor to determining poses of objects in a bin. Then, a robot arm can be directed to retrieve a selected one of the objects. The pose of an object is its 3D location and 3D orientation at the location.
One set of vision-based techniques for sensing 3D shape of surfaces assumes that the objects have non-specular surfaces, such as matte surfaces. Another set of techniques assumes that the objects have specular surfaces, such as mirror surfaces or transparent surfaces.
Non-Specular Surfaces
Computer vision-based techniques for sensing 3D shape of non-specular surfaces include structured light, time-of-flight laser scanners, stereo cameras, moving cameras, photometric stereo, shape-from-shading, and depth-from-(de)focus.
Those techniques all assume that incident light on the surface is reflected diffusely, and hence, reflected light is visible at any sensor with line-of-sight to the surface. Furthermore, many of the techniques assume that visible features are physical features with a measurable 3D physical location, and not reflected features. The techniques degrade as the surface becomes less diffuse and more specular, because the above assumptions are no longer true.
Specular Surfaces
Computer vision-based techniques for sensing 3D shape of specular surfaces assume that there are features in a surrounding scene that are reflected by the specular surface. The features may be sparse, such as specular highlights arising from point light sources in the scene, as in A. Blake and G. Brelstaff, “Geometry from specularity,” Proc ICCV, 1988. If the features are sparse, then the sensed 3D shape of the surface is also sparse. This is undesirable for many applications. For example, it is difficult to compute a reliable pose of an object when the sensed features are sparse. The problem can be ameliorated by moving the camera or features relative to the surface, but this is time-consuming.
The features can be dense, such as a dense binary-coded pattern, as in T.Bonfort et al, “General Specular Surface Triangulation,” Proc ACCV, 2006. However, there is a problem in using a dense binary-coded pattern of dense individual features when sensing varied surface shapes. The reflection of dense features in a planar specular surface such as a flat mirror is not distorted, while the reflection of the features in a curved specular surface such as a spoon can be severely distorted. If the dense features are a suitable size to be visible in a planar reflecting surface, then the same features are typically too small to discern in most of the curved reflecting surface. If the dense features are a suitable size to be visible in the curved reflecting surface, then the same features are too large to provide fine resolution measurements of a planar surface.
A second problem with using a dense binary-coded pattern is that the pattern is made by displaying a succession of images on a screen. If the binary-coding has 8-bits for example, then eight images must be displayed. If the pattern must be displayed twice, as in the method described by Bonfort referenced above, then 16 images must be displayed. This is time-consuming.
A third problem with using a dense binary-coded pattern arises when a camera pixel (or group of pixels) records the reflection of a single feature of the pattern. The recorded feature is used to determine one 3D measurement on the surface. But there is no way to assign a sub-pixel accuracy pixel position to this 3D measurement. Instead, an arbitrary decision must be made, such as assigning the center of the camera pixel (or center of a group of pixels) to the 3D measurement. This is undesirable because accuracy is lost at the measurement stage.
The method described by Bonfort et al. attempts to deal with this problem by smoothing the 3D measurements in a subsequent smoothing stage. However, this is inferior to obtaining sub-pixel accuracy pixel positions at the measurement stage, because the smoothing can eliminate important details of the surface 3D shape.
Hybrid Surfaces
There are few vision-based sensors known in the art for objects with hybrid specular-diffuse surfaces, such as brushed metal, where the surface reflects some of the incident light in a specular way, and some of the light in a diffuse way. This is because such a surface does not give a strong enough diffuse response for techniques that work with non-specular surfaces, and the surface also does not give a strong enough specular reflection for techniques that work with specular surfaces. For example, the method described by Bonfort et al. fails when the reflections of adjacent binary-coded features are blurred into each other due to the diffuse reflection component. Therefore, the binary-coded pattern cannot be determined.
Thus, there is a need for a method and system for sensing specular surfaces that performs well on both planar and curved surfaces. There is also a need for a system that is fast because the pattern that it uses is not composed of a large temporal sequence of images. There is also a need for a method and system that associates camera pixel positions with sub-pixel accuracy to the determined 3D measurements. And, there is a need for a method and system for sensing specular surfaces that can cope with the presence of a diffuse component in the surface reflectance.